👨‍💻 Professional notebooks 👨‍🚀

Hunter’s haphazard hoard of hopefully helpful hacks

Algorithms | Athena workstations at MIT | AWS | C | Catholicism | Cloud | Databases | Distributed systems | Docker | Finances | Future topics | GDB | Games | Golang | Grammar and style | Housing | IntelliJ | Interviews | Julia | Kubernetes | LaTeX | Linear algebra | Linux | Literature | Mac | Machine learning | Miscellanea | Networks | ONC RPC | Operating systems | Pandoc | Probability | Projects | Python | Raspberry Pi | Screen | Security | Serialization | Shell | Software engineering | SSH | Strength training | System design | Tmux | Version control | Vim | Virtual machines | Visual Studio Code | Visualize C call graphs on a Mac | Web programming | Wolfram Alpha

Finances

Salary information

Equity compensation

AWS

General

VPC

EC2

IAM

Kubernetes

General

API server

Scheduling

Node runtime

Scale

DNS resolution

EKS

Docker

Containers

Reference

Snippets

Runtime privilege and Linux capabilities

Default granted - AUDIT_WRITE write records to kernel auditing log - CHOWN make arbitrary changes to file UIDs and GIDs (see chown(2)) - DAC_OVERRIDE bypass file read, write, and execute permission checks - FOWNER bypass permission checks on operations that normally require the file system UID of the process to match the UID of the file - FSETID don’t clear set-user-ID and set-group-ID permission bits when a file is modified - KILL bypass permission checks for sending signals - MKNOD create special files using mknod(2) - NET_BIND_SERVICE bind a socket to internet domain privileged ports (port numbers less than 1024) - NET_RAW use RAW and PACKET sockets - SETFCAP set file capabilities - SETGID make arbitrary manipulations of process GIDs and supplementary GID list - SETPCAP modify process capabilities - SETUID make arbitrary manipulations of process UIDs - SYS_CHROOT use chroot(2), change root directory

Default not granted - AUDIT_CONTROL enable and disable kernel auditing; change auditing filter rules; retrieve auditing status and filtering rules - AUDIT_READ allow reading the audit log via multicast netlink socket - BLOCK_SUSPEND allow preventing system suspends - BPF allow creating BPF maps, loading BPF Type Format (BTF) data, retrieve JITed code of BPF programs, and more - CHECKPOINT_RESTORE allow checkpoint/restore related operations. Introduced in kernel 5.9 - DAC_READ_SEARCH bypass file read permission checks and directory read and execute permission checks - IPC_LOCK lock memory (mlock(2), mlockall(2), mmap(2), shmctl(2)) - IPC_OWNER bypass permission checks for operations on System V IPC objects - LEASE establish leases on arbitrary files (see fcntl(2)) - LINUX_IMMUTABLE set the FS_APPEND_FL and FS_IMMUTABLE_FL i-node flags - MAC_ADMIN allow MAC configuration or state changes. Implemented for the Smack LSM - MAC_OVERRIDE override Mandatory Access Control (MAC). Implemented for the Smack Linux Security Module (LSM) - NET_ADMIN perform various network-related operations - NET_BROADCAST make socket broadcasts, and listen to multicasts - PERFMON allow system performance and observability privileged operations using perf_events, i915_perf and other kernel subsystems - SYS_ADMIN perform a range of system administration operations - SYS_BOOT use reboot(2) and kexec_load(2), reboot and load a new kernel for later execution - SYS_MODULE load and unload kernel modules - SYS_NICE raise process nice value (nice(2), setpriority(2)) and change the nice value for arbitrary processes - SYS_PACCT use acct(2), switch process accounting on or off - SYS_PTRACE trace arbitrary processes using ptrace(2) - SYS_RAWIO perform I/O port operations (iopl(2) and ioperm(2)) - SYS_RESOURCE override resource Limits - SYS_TIME set system clock (settimeofday(2), stime(2), adjtimex(2)); set real-time (hardware) clock - SYS_TTY_CONFIG use vhangup(2); employ various privileged ioctl(2) operations on virtual terminals - SYSLOG perform privileged syslog(2) operations - WAKE_ALARM trigger something that will wake up the system

Tcpdump

Cloud

Prometheus

Terraform

Elasticsearch

Swagger

Interviews

Overview

Interview types

Interviews generally fall into 1 of 3 categories

Additional notes

Tools

Remote-friendly

Remote-friendly big-tech companies (as of January 2024)

Prep

Rubrics

Coding

Design

Behavioral

System design

Specific problems

Partitioning and replication

Latency

Software engineering

Overview

Decision making

Development

Collaboration

User experience

Patterns

Culture

Concurrency

Typing

CI/CD

Package management

Computation structures

Tools

Libraries

SSH

IntelliJ

Serialization

Intro

Some formats

Protocol buffers

gRPC

Canonical forms

Literature

Catholicism

Games

Traditional sports

Factorio

Scrabble

Housing

General

San Francisco

Linear algebra

Resources

Topics

Julia

Reference

Snippets

Jupyter notebooks

Strength training

General

Form

Fixes

Macro tracking

Kinesiology

Shell

Intro

General

Scripting

Snippets

# spin for the passed number of seconds, defaulting to 10 seconds.
function spin() {
  nsec=$(( ${1-10}*10 ))
  spin='-\|/'
  for (( i = 0 ; i < nsec ; i++ )) ; do
    j=$(( i % 4 ))
    printf "\r${spin:$j:1}"
    sleep .1
  done
  printf "\r \n"
}

Options

Tools

Miscellanea

Technical topics

Non-technical lists

Life hacks

Sell vehicle privately

Food

Home

Tech

Other

Grammar and style

Style

Punctuation

Disambiguations

Abbreviations

LaTeX

Overview

Intro

Miscellanea

Citations

General

Machine learning

Distributed systems

Algorithms

Resources

Algorithms

Data structures

Probability

Wolfram Alpha

Pandoc

Future topics

Raspberry Pi

Intro

Boards

Physical components

Install Raspbian

  1. Download Raspbian
  2. Unzip and copy image to SD card
  3. Safely remove SD card: sudo sync
  4. Insert SD card into Pi, power on, plug in HDMI monitor and USB peripherals, boot
  5. Edit desired config values in /boot/config.txt and reboot

Software

Fun projects

Peripherals

Hardware

Miscellanea

Linux

General

Debian

Ubuntu

Performance

Networking

Proxies

File system

systemd

Commands

Snippets

Miscellanea

Version control

General

Git

Web programming

Chrome DevTools

NPM

DOM

JavaScript

REST

HTTP

Operating systems

Reference

General

Golang

Overview

Mac

Specific apps

General

Keyboard shortcuts

Homebrew

Essential apps

Python

Intro

Installs

Style

General

Snippets

Code

import asyncio

async def main():
    """Asynchronous main function."""
    print('hello')
    await asyncio.sleep(1)
    print('world')

if __name__ == '__main__':
    asyncio.run(main())
def overrides(interface_class):
    """
    Allow runtime checking of @override() decorator.

    Usage:
    > @overrides(logging.Handler)
    > def emit(self, record):
    >     pass

    See https://stackoverflow.com/questions/1167617/#answer-8313042.
    """

    override_err = 'Overriden method does not exist in parent.'

    def overrider(method):
        assert method.__name__ in dir(interface_class), override_err
        return method

    return overrider

Virtual machines

Intro

Vagrant

Packer

Databases

General

Graph databases

SQLite

Redis

MySQL

PostgreSQL

C++

C

Intro

Language

Standards

Compilation

Preprocessor

Make

CMake

Sockets

GDB

Overview

Intro

Commands

Startup

Moving around

Looking around (tools)

Looking around (tips)

Breakpointing

Editing memory

Miscellanea

Security

Overview

Information security

Security tools

ASN.1

OpenSSL

Let’s Encrypt

Security primitives

Keying

Public-key cryptography

Physical security

Miscellanea

Networks

Intro

Topology

Routing

Proxying

Internet

Local networks

OSI model

Tools

Simulation

ONC RPC

Overview

References

Commands

Miscellanea

Athena workstations at MIT

General

How the filesystem works

Using the VPN

Useful to e.g. access academic papers released only to people on MIT’s network. - Why MIT’s VPN is great - Install the VPN software: Cisco’s AnyConnect VPN - Log in to the VPN with Duo: VPN address is vpn.mit.edu/duo

Better remote login

We want to be able to log in easily–that is, without providing a password. We’ll use the Kerberos protocol. - Install Kerberos Extras to allow creation of a Kerberos ticket - Acquire a Kerberos ticket using the Ticket Viewer app. Alternatively, use the kinit family of programs: - kinit <athena-username> get Kerberos ticket - klist list tickets - kdestroy destroy tickets - Add the following block to your ~/.ssh/config file. This will allow login with ssh athena, and will automatically use local Kerberos tickets.

```sshconfig
Host athena
    HostName athena.dialup.mit.edu
    User <athena-username>
    GSSAPIAuthentication yes
    GSSAPIDelegateCredentials yes
    StrictHostKeyChecking no  # optional, fixes an issue I kept running into
```

Better remote editing

We want to be able to edit files in our Athena workspace using the tools on our local machine. - Common file-sync/remote editing tools: SSH, SCP, rsync - File transfer overview and tools - SSHFS, generic file-transfer solution for editing files on an Athena directory from your local machine - OpenAFS is an AFS-specific alternative to SSHFS, and one officially supported by MIT - Intro - How to install OpenAFS on Mac - Install AuriStor’s implementation of AFS, as is recommended by IS&T (MIT’s cell is athena.mit.edu) - Optionally allow usage from menu bar - Log into AFS: kinit <username> && aklog - Commands that touch many files are slow over mounted remote filesystems, so if your editing environment makes calls like git status this will slow down the editing environment and make it appear sluggish. Instead, we could use rsync to copy files over automatically. Sample rsync script: - 824-sync pull copy remote files to local machine one time - 824-sync push copy local files to remote machine on changes - Make sure you performed the Better remote login steps above–this won’t work without a Kerberos ticket, etc. - May need to install fswatch, e.g. brew install fswatch

```sh
# Store this in a script such as `~/.scripts/824-sync`

# Usage: 824-sync <push/pull> [-f]

# This script syncs the `src` directory of my 6.824 work from the local machine
# to the Athena machine.

# This won't delete files on the Athena machine.
# It also won't overwrite "newer" on the receiver unless you pass `-f`.

shared_dir="src"  # shared dir across local, remote
src_dir="/Users/Hunter/Desktop/mounts/${shared_dir}"
dest_dir="/mit/hcg/6.824/${shared_dir}"
exclude="*.swp"

# Return early if no $1
[ -z "$1" ] && echo "Usage: <push/pull> [-f]" && exit 1

# Don't protect newer files on receiver when -f passed
[[ "$2" = '-f' ]] && u='' || u='u'

rsync_push="rsync -az${u}P --exclude=${exclude} ${src_dir}/ athena:${dest_dir}"
rsync_pull="rsync -az${u}P --exclude=${exclude} athena:${dest_dir}/ ${src_dir}"

# Just pull and exit
if [ "$1" = 'pull' ] ; then
    echo 'pulling...'
    $rsync_pull
    exit
fi

# Continuously push
echo 'pushing...'
$rsync_push
echo 'watching...'
fswatch -or "$src_dir" | while read ; do $rsync_push ; done
```

Projects

Twilio-Dropbox integration

EOD toy robot

Screen

Tmux

Vim

Intro

Miscellanea

UI

Motion

View

Edit

Troubleshoot

Plugins

Visual Studio Code

Visualize C call graphs on a Mac

Two-liner

Summary

Alternatives

See this Stack Overflow answer for a list of alternatives to this approach.

Compiling with RTL representation

brew install gcc
gcc-7 -fdump-rtl-expand <*.c>

As a precursor, egypt wants to use output from GCC, while, apparently, Macs install clang and call it gcc. To fix this, brew install gcc and use gcc-7 instead of gcc.

RTL representation, the Register Transfer Language, is a low-level intermediate representation used during the last part of compiler work. Egypt uses this representation to create a graph-based call graph. The -fdump-rtl-expand flag dumps the RTL representation for Egypt to use.

Installing graphviz and egypt

brew install graphviz
# Download egypt from http://www.gson.org/egypt/
cd <download-location>
perl Makefile.PL
make && make install

Generate visual

perl <path/to/egypt> <generated-rtl-file> | dot -Tpng -o out.png
open out.png

Dot is a directed graph visualizer. It takes the graph-specification output from egypt and here outputs a PNG.